1 link tagged with all of: bot detection + agent traffic + mintlify + ai crawlers
Click any tag below to further narrow down your results
Links
Mintlify built agent-traffic, a dataset and small classifier that tells AI agents, crawlers, bots and people apart from web requests. It checks claimed identities against operators' published IP ranges to catch impersonators.
- Agent traffic to documentation sites is growing fast, with coding agents like Claude Code and Cursor fetching docs for users, alongside AI training crawlers.
- The classifier verifies crawlers like GPTBot against OpenAI's published IP ranges and flags requests that claim a crawler identity from the wrong address.
- Agents that disguise themselves with a browser user agent can still be caught by an Accept header that prefers markdown over HTML, which browsers never send.
- Some signals are blind spots: agentic browsers running inside a person's own Chrome look identical to a human, and logs that keep only the user agent miss the header check.